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Introduction 


This  project  report  is  the  summary  of  a  three-phase  initiative  to  install  Picture  Archiving  and 
Communication  Systems  (PACS)  and  Teleradiology  at  North  Mississippi  Medical  Center’s 
(NMMC)  main  hospital  campus  in  Tupelo,  four  community  hospitals  in  northeastern  Mississippi 
and  one  community  hospital  in  northwestern  Alabama,  and  at  15  clinics  and  to  test  relevant 
research  hypotheses  pertaining  to  quality,  security,  and  clinical  issues. 

Phase  I  deliverables  included  1)  installation  of  PACS  equipment  secured  with  Army  Security 
Architecture  for  Medical  (ARSAM)  Systems  Design  Plan  on  all  freestanding  buildings  on  the 
NMMC  campus,  including  NMMC-Tupelo,  Women’s  Health  Center,  Longtown  Imaging, 
Digestive  Health,  Internal  Medicine  Associates,  and  the  Cancer  Center,  2)  training  of  NMMC’ s 
radiologists,  3)  design  of  research  database,  and  4)  collection  and  analysis  of  efficiency, 
satisfaction,  and  outcome  data  regarding  the  NMMC-Tupelo  campus. 

Phase  II  deliverables  included  1)  installation  of  PACS  equipment  secured  with  ARSAM  Systems 
Design  Plan  at  five*  community  hospitals  (four  in  northeastern  Mississippi  and  one  in 
northwestern  Alabama),  2)  training  of  all  radiology  staff  members  at  these  facilities,  3)  multiple 
tests  of  security  architecture,  and  4)  collection  and  analysis  of  efficiency,  satisfaction,  and 
outcome  data  regarding  the  community  hospitals.  *We  encountered  technical  across-state-line 
difficulties  with  implementing  PACS  in  the  hospital  located  in  Hamilton  Alabama,  so  only  four 
Mississippi  community  hospitals  were  implemented  during  the  Phase-11  timeline  ( October  2005  - 
September  2006).  PACS,  however,  was  implemented  in  the  fifth  community  hospital  in  June 
2008.  Please  see  “Project  Deviations”  for  a  full  explanation. 

Phase  III  deliverables  include  1)  installation  of  PACS  equipment  secured  with  ARSAM  Systems 
Design  Plan  at  15  clinics,  2)  training  of  radiology  staff  members  at  these  facilities,  3)  multiple 
tests  of  security  architecture,  and  4)  collection  and  analysis  of  efficiency,  satisfaction,  and 
outcome  data  regarding  the  clinics. 

This  final  report  provides  1)  an  overview  of  the  impact  of  the  efficiency  of  radiologists  and  other 
medical  personnel  compared  to  baseline,  2)  the  satisfaction  of  physician  providers  compared  to 
baseline,  3)  the  efficacy  of  primary  care  providers  providing  the  initial  interpretation  of 
radiological  studies  -  prior  to  the  implementation  of  PACS,  and  4)  an  assessment  of  ARSAM 
penetration  to  assess  resistance  to  intrusion  and  device  compromise. 

Body 


In  2004,  NMMC,  in  conjunction  with  its  health  system,  North  Mississippi  Health  Services 
(NMHS),  proposed  a  three-phase  initiative  to  install  Picture  Archiving  and  Communication 
Systems  (PACS)  and  Teleradiology  at  its  main  hospital  campus,  five  community  hospitals  in 
northeastern  Mississippi,  one  community  hospital  in  northwestern  Alabama,  and  at  15  clinics.  In 
addition  to  improving  the  quality  of  radiology  services  for  people  living  in  this  rural  region, 
including  military  personnel,  military  reserve  personnel,  and  military  dependents,  the  PACS 
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systems  would  be  used  to  test  security  architecture  for  networked  medical  devices,  an  initiative 
of  critical  importance  to  the  military,  and  to  research  clinical  outcomes  associated  with  PACS 
technology  that  are  of  significance  to  the  military  and  other  health  care  operations. 

The  hypotheses  to  be  tested  were  that  implementation  of  state-of-the-art  PACS  on  the  NMMC- 
Tupelo  campus,  five  community  hospitals,  and  other  sites  affiliated  with  the  NMHS  system 
will  1)  improve  radiology  efficiency,  2)  improve  clinical  outcomes,  and  3)  demonstrate  that 
networked  PACS  can  be  made  secure  to  intrusion  and  device  compromise. 

Expected  results  of  the  initiative  included: 

1 )  Increased  efficiency  of  radiologists  as  measured  by  performance  indicators 

2)  Decreased  number  of  radiology  support  staff  (film  librarians) 

3)  Increased  satisfaction  of  medical  providers  as  measured  by  satisfaction  surveys. 

4)  Improved  treatment  plans  for  clinic  patients  who  undergo  a  CXR. 

5)  Improved  clinical  outcomes  as  measured  by  specific  radiology  indicators  on  the  length  of 
stay  (LOS)  and  other  indicators  for  CAP  and  stroke 

6)  Identification  of  highly  effective  security  architecture  for  networked  PACS  systems  that 
can  be  replicated  for  Army  Medicine’s  networked  medical  devices,  including  PACS 

Potential  benefits  of  the  three-phase  project  included: 

•  Demonstration  of  radiology  service  management  at  remote  locations 

•  Demonstration  of  radiologists’  efficiency  in  interpreting  tests  at  remote  locations. 

•  Information  on  the  impact  of  PACS  technology  on  clinical  outcomes  for  hospitalized 
patients  with  CAP  and  stroke 

•  Information  on  the  impact  of  PACS  technology  on  treatment  planning  for  clinic  patients  who 
undergo  CXR. 

•  Demonstration  of  secure  teleradiology  services  at  remote  locations 
The  project’s  relevance  to  the  military  is  related  to  its  capacity  to: 

•  Simulate  the  military’s  management  of  radiology  services  in  remote,  noncombat  settings 

•  Validate  ARSAM  security  architecture  for  PACS 

•  Ensure  availability  of  high  quality  radiological  services  for  military  personnel,  reserve 
personnel,  and  dependents  living  in  northeastern  Mississippi. 

Project  Period  Deliverables  and  Results 

Deliverables  for  this  project  were  met.  During  the  period  from  October  1,  2004  to  June  30,  2008: 

•  PACS  equipment  became  fully  operational  at  the  NMMC-Tupelo  campus  (including 
NMMC-Tupelo,  Women’s  Health  Center,  Longtown  Imaging,  Digestive  Health,  Internal 
Medicine  Associates,  and  the  Cancer  Center),  all  five  community  hospitals  (Eupora, 
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Pontotoc,  West  Point,  Iuka  and  Hamilton,  AL  sites),  and  at  15  primary  care  clinics  in  rural 
Mississippi  (Pulmonary  Clinic,  Family  Medicine  Residency  Center,  Barnes  Crossing,  Med 
Serve,  Eupora  FMC,  Okolona  FMC,  North  Columbus  FMC,  Fowndes  FMC,  Pontotoc  FMC, 
Fulton  FMC,  Baldwyn  FMC,  Oxford  FMC,  Chickasaw  FMC,  Hamilton  Medical  Clinic  and 
Saltillo  FMC 

•  Radiology  staff  at  all  sites  were  thoroughly  trained 

•  Data  were  collected,  analyzed  and  reported  (summary  below) 

•  Network  vulnerability  assessment  and  penetration  tests  were  completed  with  results 
reported;  Network  vulnerability  recommendations  were  installed  and  assessment  and 
penetration  tests  were  completed  with  results  reported. 

Key  Research  Accomplishments 

We  report  our  research  accomplishments  in  each  of  the  five  research  areas  (Efficiency,  Cost, 
Satisfaction,  Clinical  Outcomes  and  Security  Architecture  Testing).  As  appropriate,  we  report 
research  area  results  for  each  of  the  three  phases  and  a  post-phase-3  assessment  (conducted  April 
through  June  2008)  which  presents  an  overview  of  the  project’s  impact. 

■  Research  Area:  Impact  of  PACS  on  efficiency  of  radiologists  as  measured  by 
performance  indicators 

In  all  three  phases,  dramatic  reductions  were  noted  in  Report  Turnaround  Time  from  pre-  to 
post-PACS  implementation.  Report  Turnaround  Time  is  the  time  that  elapses  from  the 
completion  of  the  procedure  until  the  radiologist’s  interpretation  of  the  image  is  available  for  the 
clinician’s  review.  Report  Turnaround  Time  is  the  key  measure  for  improved  efficiency  because 
it  determines  when  the  radiological  study  can  be  useful  in  clinical  decision  making. 

Other  measures  of  efficiency  include  the  radiologists’  productivity ,  specifically  how  many  more 
films  they  can  interpret  because  they  don't  have  to  spend  time  traveling  in  order  to  interpret 
them.  Prior  to  the  implementation  of  PACS,  the  emergency  department  physicians  would 
interpret  their  own  films  and  initiate  treatment  accordingly.  The  radiologists  would  also  interpret 
the  films  and  the  variances  of  interpretations  have  been  noted.  The  number  of  lost  films  is  also 
important  not  only  are  lost  films  a  lost  opportunity  to  compare  a  current  film  with  a  historical 
film,  it  also  takes  significant  time  to  look  for  the  lost  film. 

Phase-1  NMMC-Tupelo  Campus 

The  baseline  data  for  these  comparisons  are  the  12  months  prior  to  the  implementation  of  PACS 
in  November  2004  and  the  follow-up  period  is  the  12  months  post-PACS  implementation. 

■  Report  Turnaround  Time  decreased  from  an  average  time  of  2.27  hours  (11/2003  through 
10/2004)  to  1.04  hours  in  the  following  12  months 

■  Radiologists'  productivity  increased  by  9.1% 

■  Projected  savings  -  $518,430  in  first  12  months  of  PACS. 
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■  Emergency  Services  Department  (ESD)  interpretation  variance  decreased  from  106 
(11/2003  through  10/2004)  to  ZERO  in  the  following  12  months. 

■  Lost  films  decreased  from  230  (1 1/2003  through  10/2004)  to  ZERO  in  the  following  12 
months. 

Phase-2:  Community  Hospitals 

•  At  the  hospitals  in  Eupora  and  Pontotoc,  the  wait  for  interpretations  decreased  from 
averages  of  17  hours  to  less  than  two  hours  (Refer  to  Figure  1). 


Figure  1:  Report  Turnaround  Time 

(From  completion  of  procedures  to  radiologists’  interpretations) 

(March  2004  thru  August  2004  versus  March  2005  thru  August  2005) 


Turnaround  Time 

Turnaround  Time 

(March  2004  thru 

(March  2005  thru 

August  2004) 

August  2005) 

Pontotoc 

13  hours 

.5  hours 

West  Point 

9  hours 

1.8  hours 

Eupora 

17  hours 

1.3  hours 

Iuka 

17  hours 

1.8  hours 

Hamilton** 

**  The  NMMC-Hamilton  Hospital  is  located  in  Hamilton  AL.  Due  to  technical  issues  involving 
transmission  charges  across  state  lines,  it  was  not  possible  to  implement  PACS  during  Phase-2  as 
originally  planned.  PACS,  however,  was  implemented  at  NMMC-Hamilton  in  June  2008,  but 
we  cannot  yet  report  on  its  efficacy.  The  technical  issue  and  its  resolution  are  described  in  the 
“Project  Deviations”  section. 

Note:  The  Phase-2  cost  savings  in  the  community  hospitals  are  described  in  Figure-4. 

Phase-3:  Clinics 

•  We  randomly  selected  60  patients  who  received  a  CXR  at  one  of  the  clinics  prior  to  and 
after  the  implementation  of  PACS.  The  pre-PACS  timeframe  was  January  2006  and  the 
post-PACS  timeframe  was  December  2006.  Prior  to  the  implementation  of  PACs  in  the 
clinics,  an  X-ray  would  be  performed  in  the  clinics  and  then  transported  by  courier  to  the 
radiologist  in  Tupelo  (a  distance  of  over  90  miles  for  the  most  remote  clinic).  The 
Tupelo-based  radiologist  would  interpret  the  study  and  send  the  interpretation  via  the 
electronic  medical  record  (EMR)  that  links  the  hospitals  and  clinics.  A  chart  review  of 
the  EMR  was  performed  and  the  following  elements  were  collected  to  examine  the 
efficiency  of  the  PACS  system  in  the  clinics  (see  Figure  2  for  results): 

>  When  the  patient  visit  occurred  (dale  and  time  -  in  minutes)-, 

'r  When  the  radiologist’s  interpretation  was  available  in  the  EMR; 

>  When  the  PCP’s  signature  appeared  on  the  radiologist’s  interpretation  (to  indicate 
the  PCP  had  seen  the  radiologist’s  interpretation);  and 

>  When  the  final  primary  care  provider  (PCP)  signature  appeared  on  the  overall 
visit  (to  indicate  the  PCP’s  closure  on  this  episode ). 
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A  2-sample  t-test,  adjusted  for  unequal  variance,  where  appropriate,  was  used  to  calculate  the  p- 
values. 


Figure  2:  Pre  and  Post-PACS  efficiency  in  NMHS  clinic  setting  for  the  CXRs  that  were 
studied  in  the  random  sample. _ _ _ _ 


Pre-PACS  (hrs) 

Post-PACS 

(hrs) 

p-value 

Radiologist’s  interpretation  available  on 
EMR 

49 

1.6 

<.0001 

PCP  signs  off  on  radiologist’s 
interpretation  in  EMR 

44 

35 

.2406 

PCP  final  sign  off  on  visit 

90 

37 

<.0001 

It  appears  that  the  radiologists  have  significantly  reduced  the  turnaround  time  for  reporting  their 
interpretations  on  the  patient’s  EMR.  The  PCPs,  however,  have  not  significantly  reduced  the 
time  before  signing  off  on  having  read  the  radiologists’  interpretations.  The  PCP’s  final  sign  off 
on  the  patient’s  chart,  however,  did  reduce  significantly.  Note,  the  clinical  impact  of  this 
assessment  is  reported  in  Figure  11. 

Post-Phase-3  Assessment  -  A  Review  of  the  PACS  Project  Conducted  in  April- June  2008 
A.  Clinic  PACS  Efficiency  Assessment 

As  noted  in  Phase-3  assessment  above,  there  was  a  dramatic  difference  in  the  Report  Turnaround 
Time,  but  based  on  the  EMR  timeline,  the  PCPs  did  not  appear  to  incorporate  this  new 
information  into  their  clinical  decision-making.  We  performed  another  pre-  and  post-PACS 
implementation  in  two  rural  clinics,  different  than  the  two  clinics  analyzed  in  Phase-3.  .  We 
randomly  sampled  20  patients  who  received  CXRs  in  each  of  the  two  clinics,  both  before 
(January  2006)  and  after  (October  2007)  PACS  implementation.  The  efficiency  changes  are 
reported  in  Figure  3  and  the  clinical  impact  of  this  assessment  is  reported  in  Figurel3. 


Figure  3:  Pre  and  Post-PACS  efficiency  in  NMHS  clinic  setting 


Pre-PACS 
Clinic- 1  (hrs) 

Post-PACS 
Clinic- 1  (hrs) 

Pre-PACS 
Clinic-2  (hrs) 

Post-PACS 

Clinic-2 

(hrs) 

Radiologist’s  interpretation 
available  on  EMR 

81.5 

1.4 

43.2 

2.4 

PCP  signs  off  on  radiologist’s 
interpretation  in  EMR 

53.2 

2.1 

98.3 

120.6 

PCP  final  sign  off  on  visit 

108.3 

3.3 

101.8 

32.6 

B.  Overall  Efficiency  Assessment 

The  following  report  on  the  impact  of  PACS  on  several  measures  of  efficiency  was  produced  by 
Gordon  Hollingsworth,  the  Director  of  Radiology  at  NMMC,  who  has  been  integral  to  the 
development  and  implementation  of  PACS  and  has  performed  the  efficiency  assessments  over 
the  first  three  phases  of  this  project.  The  time  period  of  the  noted  changes  is  from  pre-PACS 
(9/04)  to  post-PACS  (6/08)  and  covers  all  three  phases  of  PACS  implementation  (NMMC- 
Tupelo  campus,  community  hospitals  and  the  rural  community  clinics). 

1.  Report  Turnaround  Time  for  the  entire  system  has  experienced  a  65%  reduction  (2.27 
hours  to  0.8  hours). 

2.  The  specific  report  turnaround  times  for  all  of  the  hospitals  during  the  phases  of  this 
project  are  reported  in  Figure  13. 

3.  Number  of  film  interpretations  per  Radiologist  -  a  15%  increase  of  volume  per 
Radiologist  has  been  experienced.  This  would  translate  to  an  improvement  of 
approximately  one  FTE  for  this  radiology  group.  In  Phase- 1,  we  described  a  9% 
improvement  in  radiologist  efficiency.  Radiologists’  efficiency  has  continued  to 
increase  because  they  no  longer  have  to  travel  to  the  community  hospitals  to  interpret 
studies.  In  addition,  the  overall  volume  of  radiological  studies  has  increased  by  about 
15%  from  2004  until  2008  and  the  radiologist  group  has  been  able  to  provide  full 
coverage  with  the  same  FTEs. 

4.  The  number  of  ESD  physician  first  interpretation  vs.  radiologist  interpretation  has 
decreased  to  ZERO. 

5.  Number  of  lost  films  -  lost  films  has  decreased  to  ZERO. 

►  Research  Area:  Impact  of  PACS  on  costs  for  radiology  personnel,  film  reduction 
and/or  decreased  courier  usage. 

Phase-1:  NMMC-Tupelo  Campus 

■  Film  librarian  personnel  decreased  by  30% 

■  Projected  savings  -  $300,000  per  year  starting  in  November  2004  and  going 
forward 

■  Reduction  in  film  usage 

■  Projected  savings  -  $450,000  per  year  starting  in  November  2004  and  going 
forward 

Phase-2:  Community  Hospitals 

•  All  four  hospitals  experienced  cost  savings  as  a  result  of  PACS  implementation.  All 

hospitals  reduced  their  film  costs,  but  only  two  reduced  their  personnel  costs.  Radiology 
technicians  at  Pontotoc  and  Eupora  also  doubled  as  film  librarians,  so  their  positions  were 
maintained  when  PACS  was  introduced.  (Figure-4) 
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Figure  4:  Cost  Reduction 


(February  2005  through  January  2006) 


Film  Cost  Reduction 

Personnel  Reduction/ 
Salary  Savings 

Pontotoc 

$14,000 

No  change 

West  Point 

$14,100 

$10,729 

Eupora 

$26,361 

No  change 

Iuka 

$43,000 

$11,000 

Phase-3:  Clinics 

The  NMMCI  administrators  performed  this  evaluation.  They  noted  a  decrease  in  film  costs,  but 
an  overall  increase  in  their  MIS  support  fees  and  increased  bandwidth  costs.  These  increases 
more  than  outweigh  the  film  reduction  savings  and  since  January  2007,  their  radiology  costs  are 
up  by  22%  since  the  implementation  of  PACS.  There  is  no  decrease  in  courier  costs,  since  the 
courier  is  still  transporting  other  materials  between  NMMC-Tupelo  and  the  clinics. 

Post-Phase-3  Assessment  -  A  Review  of  the  PACS  Project  Conducted  in  April-June  2008 

An  extension  of  Mr.  Hollingsworth’s  overall  PACS  efficiency  analysis  includes  the  following 
financial  considerations: 

1.  Number  of  film  librarians  -  although  the  volume  of  radiological  studies  continues  to  rise, 
we  have  experienced  a  reduction  in  film  librarian  hours  by  20,000  hours  per  year.  This 
amounts  to  a  savings  of  $300,000  per  year  over  each  of  the  last  four  years.  The  number  of 
film  librarians  has  remained  stable  after  the  reduction  in  Phase- 1.  These  film  librarians 
are  responsible  for  managing  historical  films.  The  number  of  film  librarians  may 
decrease  in  time,  but  it  will  take  a  number  of  years. 

2.  Film  costs  are  reduced  by  $600,000  per  year  at  NMMC-Tupelo,  compared  to  pre-PACS 
film  costs.  In  Phase- 1,  we  recognized  a  $450,000  savings  in  NMMC-Tupelo’ s  film 
savings  and  expected  that  savings  to  be  annual.  Since  that  time,  however,  more  units  and 
physicians  on  the  Tupelo  campus  have  converted  to  PACS  and  the  film  savings  have 
increased.  NMMC-Tupelo  does  not  have  increased  MIS  support  fees  or  bandwidth  costs 
because  of  their  main-campus  location.  The  annual  film  savings  of  $600,000  is  genuine. 

►  Research  Area:  Impact  of  PACS  on  satisfaction  of  primary  care  provides  (clinic- 
based  physicians  and  nurse  practitioners). 

Phase-2:  Community  Hospitals 
a.  Referring  Physician  Survey 

This  is  a  5 -point  survey  with  the  referring  physician  marking  5  as  being  excellent  and  1  as  being 
poor.  Their  results  are  compiled  and  compared  to  the  baseline  survey  that  was  conducted  from 
PCPs  from  the  entire  region.  Note,  the  original  pre-PACS  questions  primarily  referred  to 
radiological  studies  being  performed  at  the  hospitals.  In  each  of  the  four  categories  of 
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satisfaction  questions,  the  cumulative  score  increased  from  the  baseline  survey  in  2003  to  post- 
PACS  in  2006.  Refer  to  the  2006  column  in  Figure  5. 


•  In  each  of  the  four  categories  of  satisfaction  questions,  the  cumulative  score  increased 
from  the  baseline  survey  in  2003  to  post-PACS  in  2006.  Refer  to  the  2006  column  of 
Figure  5. 

•  Physician  response  rate  increased  from  11.8  percent  in  2003  to  33.9  percent  in  2006 

Phase-3:  Clinics 

In  June  2007,  25  PCPs  in  the  13  clinics  that  have  implemented  PACS  returned  the  survey  (the 
remaining  two  clinics  implemented  PACS  after  this  survey  was  conducted).  This  is  a  5-point 
survey  with  the  PCP  marking  5  as  being  excellent  and  1  as  being  poor.  Their  results  are  compiled 
and  compared  to  the  baseline  survey  that  was  conducted  from  PCPs  from  the  entire  region.  Note, 
the  original  pre-PACS  questions  primarily  referred  to  radiological  studies  being  performed  at  the 
hospitals.  In  each  of  the  four  categories  of  satisfaction  questions,  the  cumulative  score  increased 
from  the  baseline  survey  in  2003  to  post-PACS  in  2007.  Refer  to  the  2007  column  of  Figure  5. 

Post-Phase-3  Assessment  -  A  Review  of  the  PACS  Project  Conducted  in  April- June  2008 

Physician  Provider  Satisfaction  Surveys 

The  same  survey  tool  was  used  as  before,  but  instead  of  asking  specific  populations  of  physicians 
regarding  their  satisfaction  with  PACS,  we  e-mailed  surveys  to  all  of  the  physicians  (340)  who 
have  staff  privileges  at  an  NMHS  facility.  Twenty- two  physicians  returned  the  survey. 

This  is  a  5-point  survey  that  asks  the  physician  to  mark  5  as  being  excellent  and  1  as  being  poor. 
The  2008  survey  results  are  compiled  and  compared  to  the  baseline  survey  that  was  conducted  in 
2003,  the  referring  community  hospital  physicians  in  2006  and  clinic’s  PCPs  in  2007.  Whereas 
all  post-PACS  surveys  show  an  improvement  in  physician  satisfaction,  the  highest  satisfaction 
occurred  in  2006  and  reflects  PACS  being  implemented  in  the  clinics. 

Figure  5:  Summary  of  PACS  Physician  Surveys 


Referring  Physicians/Clinic-based  PCPs 

2003 

2006 

2007 

2008 

Scheduling  &  Registration 

3.65 

4.33 

4.42 

4.23 

Time  it  takes  to  schedule  outpatients 

4.25 

4.40 

4.15 

Responsiveness  for  urgent  patients 

4.42 

4.44 

4.30 

Physician  Needs 

3.74 

4.15 

4.66 

4.19 

Availability  of  radiologists  to  review  films 

4.36 

4.76 

3.93 

Timeliness  of  receiving  preliminary  reports 

3.97 

4.60 

3.93 

Timeliness  of  receiving  final  reports 

4.01 

4.72 

4.07 
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Timeliness  in  completion  of  urgent  exams 

Report  turn-around  time  for  urgent  patients 

4.29 

4.14 

4.76 

4.64 

4.50 

4.54 

Patient  Needs 

3.72 

3.97 

4.43 

3.94 

Waiting  time  in  the  department 

3.57 

4.43 

3.90 

Customer  service  attitude  &  tech  performance 

4.37 

4.43 

4.00 

Radiologists 

3.93 

4.16 

4.48 

4.12 

Quality  &  accuracy  of  interpretation 

4.20 

4.74 

3.86 

Availability  for  consultation 

4.15 

4.64 

4.21 

Calling  of  positive  reports  to  physicians 

4.15 

4.08 

4.29 

To  supplement  the  empirical  data,  which  trends  strongly  to  greater  satisfaction  with  radiology 
services,  PACS  physician  survey  respondents  write-in  comments  are  presented  in  Figure  6. 

Figure  6:  Physician  write-in  comments  on  2008  regional  PACS  survey 
Do  you  feel  that  the  implementation  of  PACS  has  changed  the  way  you  treat  patients?  For  example, 
if  the  turn  around  time  for  reports  is  faster,  has  this  helped  with  your  ability  to  diagnose  and  treat 
patients? _ 

•  Turn  around  time  for  plain  films  slow. 

•  Has  helped  some. 

•  I  try  to  read  my  own  and  usually  the  patient  is  gone  before  report  is  generated. 

•  Very  helpful  to  be  able  to  view  and  compare  films  not  just  reports  even  from  my  home. 

•  I  like  the  imaging  system  for  many  reasons:  1)  the  over-reads  are  more  timely  and  sometimes  before  the 
patient  leaves  the  office,  but  not  always;  2)  PACS  lends  itself  to  conferencing  with  consultants  as  well 
as  the  radiologists;  3)  the  ability  of  reviewing  the  x-rays  with  the  patient  in  the  exam  room  is  an  added 
benefit,  as  well;  and  4)  the  ability  to  review  x-rays  performed  at  multiple  locations  is  a  great  benefit,  as 
well. 

•  Since  we  have  to  see  the  study  more  than  report,  PACS  has  dramatically  helped  our  patient  care.  We 
save  time  looking  for  the  films.  More  than  one  physician  can  review  at  the  same  time. 

•  Yes,  it  allows  me  to  view  the  film  instead  of  waiting  on  a  radiologist.  It  allows  me  to  better  interpret 
ultrasound  findings  as  the  report  given  by  the  radiologist  is  often  sub-quality. 

•  Yes,  I  am  more  impressed  with  the  quality  than  the  speed. 

•  Yes  -  the  ability  to  receive  a  report  in  a  timely  fashion  has  led  to  my  being  willing  to  wait  for  the 
radiologist’s  report  rather  than  make  treatment  plans  based  on  my  preliminary  reading. 

•  PACS  is  great  for  viewing  own  X-rays. 

•  Yes,  improved  diagnostic  studies,  turnaround  time  and  consultations. _ 


►  Research  Area:  Improved  clinical  outcomes 
Phase-1:  NMMC-Tupelo  Campus 

Two  clinical  conditions  for  which  early  and  efficient  radiological  results  are  critical  for  diagnosis 
and  treatment  and  that  present  at  relatively  high  base  rates  in  patient  populations  were  selected 
for  study:  community-acquired  pneumonia  (CAP)  and  ischemic  stroke. 
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Community  Acquired  Pneumonia:  In  our  initial  review,  we  selectively  examined  the  pre-  and 
post-  PACS  CAP  population  for  two  well-established  clinical  outcomes:  1)  Length  of  Stay 
(LOS),  and  2)  Mortality.  Because  CAP  has  a  seasonal  component,  our  selection  criteria 
included  the  same  three-month  time  periods  (Oct-Dee)  for  before-  and  after-PACS.  In  order  to 
assess  a  patient  population  in  which  we  felt  that  PACS  improvements  (timeliness  of  reports) 
would  have  the  most  impact  on  treatment  and  outcome,  we  selected  a  subpopulation  of  patients 
with  CAP  namely,  those  that  were  admitted  directly  to  the  Emergency  Services  Department 
(ESD)  rather  than  those  transferred  in  or  treated  by  their  physicians  prior  to  admission  through 
the  ESD.  Results  appear  in  Figure  7.  Whereas  the  time  to  antibiotic  administration  decreased 
slightly  (205  minutes  to  195  minutes)  it  would  be  surprising  if  the  10  minute  difference  was 
related  to  the  improvement  in  mortality  (18.2%  to  10.5%,  NS).  (Figure  7) 


Figure-7:  Community- Acquired  Pneumonia  (CAP) 


Pre-PACS 

Post-PACS 

Oct-Dec/03 

Oct-Dec/04 

No.  Patients 

44 

57 

Antibiotic  time  (min)  205 

195 

NS 

BldCulture  time 

183.5 

233.2 

NS 

(min) 

LOS  (days) 

6.4 

6.3 

NS 

Mortality 

18.2% 

10.5% 

NS 

Cost  of  care 

$9,514 

$9,361 

NS 

Ischemic  Stroke:  The  implementation  of  PACS  improved  the  interpretation  of  the  CAT  scans 
from  38  to  27  minutes,  but  it  did  not  appear  to  change  the  stroke  outcomes.  (Figure-8) 

Figure-8:  Ischemic  Stroke 


Pre-PACS 

Post-PACS 

Dec-Oct  03 

Dec-Oct  04 

No.  Patients 

55 

62 

CAT  Scan 

38  min 

27  min 

Interpretation 

13 


LOS  (days) 


5.9  5.9  NS 

Mortality  5.5%  8.1%  NS 

Phase-2  Community  Hospitals 

At  its  community  hospitals  in  Eupora,  Pontotoc,  and  West  Point,  NMMC  is  monitoring  clinical 
outcomes  as  they  relate  to  the  implementation  of  P ACS  for  two  conditions:  community- acquired 
pneumonia  (CAP)  and  ischemic  stroke,  which  were  selected  due  to  their  high  base  rates  of 
incidence  among  admissions  to  community  hospitals  and  because  diagnosis  and  initial  treatments 
of  CAP  and  ischemic  stroke  are  dependent  upon  radiological  studies. 

Reviewers  of  the  Phase  1  portion  of  this  project  (implementing  PACS  on  the  NMMC-Tupelo 
campus)  recommended  incorporation  of  process  indicators  that  were  relatively  close  to  and 
dependent  on  PACS  to  more  reliably  assess  its  impact.  The  timing  of  the  first  dose  of  antibiotic 
was  selected  as  the  process  indicator  for  CAP  and  the  administration  of  tissue  plasminogen 
activator  (tPA)  as  the  process  indicator  for  ischemic  stroke.  As  in  Phase  1,  overall  outcome 
indicators  (length  of  stay,  mortality  and  cost)  were  also  measured  pre-  and  post-PACS 
implementation 

Refer  to  Figure  9:  NMMC-Eupora  had  the  most  patients  diagnosed  with  CAP  pre-  and  post- 
PACS  (even  with  a  29%  decline  in  the  post-intervention  period).  This  patient  population  also 
was  the  most  similar  (age  and  severity  ranking)  between  the  two  assessment  periods.  During  this 
timeframe,  the  radiologists’  turnaround  times  dropped  from  17  hours  to  1.3  hours  (88  minutes), 
and  the  median  time  from  emergency  department  admission  to  administering  antibiotics  dropped 
from  176  to  98  minutes.  This  is  well  below  the  four  hour  (240  minute)  goal  for  antibiotic 
administration  and  may  have  contributed  to  the  slight  decrease  in  LOS,  5.81  to  5.47  days  (5.8%) 
and  the  significant  decrease  in  mortality  (3.96%  to  1.39%  (65%  decrease)).  The  median  cost  of 
the  admission  also  dropped  slightly,  from  $5,407  to  $4,962  (8.2%). 

The  other  two  hospitals,  West  Point  and  Pontotoc,  also  had  positive  LOS,  mortality  and  cost 
outcomes,  but  their  patient  populations  were  different  (age  and  severity)  between  the  two  time 
periods  and  their  change  in  median  antibiotic  administration  time  was  not  significantly  different. 
West  Point  did  not  demonstrate  any  change  in  its  median  time  to  antibiotic  administration.  This 
facility  has  regular  daytime  on-site  radiologist  coverage  and  experienced  the  least  improvement 
in  radiologist  study  tum-around  time  (9.0  to  1.8  hours:  Figure  1).  The  regular  presence  of  a 
radiologist  may  account  for  the  lack  of  improvement  on  antibiotic  administration,  i.e.,  the 
availability  of  PACS  did  not  change  their  practice  sufficiently.  In  short,  it  is  difficult  to  project  if 
the  implementation  of  PACS  may  have  been  associated  with  the  positive  outcomes  at  West 
Point. 
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Figure  9:  CAP  Outcome  Analysis  Table 


Median  Ave. 

Ave.  time  to  length  of 


Number 

pt 

Ave.  pt 

antibiotic 

stay 

Patient 

Median 

EUPORA 

of  pts. 

age 

severity 

(mins) 

(days) 

mortality 

Cost 

Pre-PACS  (April-Sept  2004) 

101 

67 

2.09 

176 

5.81 

3.96% 

$5,407 

Post-PACS  (April-Sept  2005) 

72 

61 

2.04 

98 

5.47 

1.39% 

$4,962 

PONTOTOC 


Pre-PACS  (April-Sept  2004) 

17 

72 

1.81 

120 

4.29 

0.00% 

$5,244 

Post-PACS  (April-Sept  2005) 

37 

65 

2.19 

115 

3.84 

0.00% 

$5,538 

WEST  POINT 


Pre-PACS  (April-Sept  2004) 

51 

66 

2.08 

120 

6.67 

7.84% 

$5,611 

Post-PACS  (April-Sept  2005) 

45 

49 

1.91 

120 

4.13 

0.00% 

$4,754 

Refer  to  Figure  10:  Only  one  hospital,  West  Point,  administered  tPA  to  one  patient.  It  was 
during  the  post-PACS  timeframe,  but  as  noted  earlier,  West  Point  already  had  about  40  hours  per 
week  of  on-site  radiologist  time,  so  it  is  impossible  to  attribute  this  tPA  usage  to  the 
implementation  of  PACS.  Although  the  numbers  of  stroke  patients  are  low,  based  on  this  6- 
month  pre-  and  post-  PACS  implementation  analysis,  it  does  not  appear  that  PACS  has  made  an 
impact  on  the  care  of  stroke  patients  with  regard  to  tPA  administration,  the  chosen  process 
marker.  Whereas  mortality  decreased  in  all  three  hospitals,  the  LOS  and  median  costs  of  care 
decreased  in  the  two  hospitals  that  did  not  administer  the  tPA. 


Figure  10:  Stroke  Outcome  Analysis  Table 

Ave. 

Number  pt 
of  pts.  age 

EUPORA 


Ave. 

tPA 

Ave. 

length 

Patient 

sev 

adminis¬ 

of  stay 

mortality 

Median 

ind 

tration 

(days) 

Cost 

Pre-PACS  (April-Sept  2004) 

16 

79 

2.25 

0 

5.69 

6.30% 

$4,482 

Post-PACS  (April-Sept  2005) 

15 

82 

2 

0 

3.56 

0.00% 

$3,487 

ILKA 


Pre-PACS  (July-Dee  2004) 

4 

71 

2.5 

0 

3.75 

25.00% 

$6,856 

Post-PACS  (July-Dee  2005) 

13 

69 

2.5 

0 

3.93 

7.10% 

$4,080 

WEST  POINT 


Pre-PACS  (April-Sept  2004) 

16 

74 

2.38 

0 

5.00 

6.30% 

$5,169 

Post-PACS  (April-Sept  2005) 

19 

68 

2.11 

1 

6.47 

0.00% 

$6,547 
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Phase-3:  Clinics 

During  the  randomly- selected  60  patient  pre-  and  post-PACS  chart  review  (described  in  Figure  2) 
two  more  data  elements  were  collected  to  examine  the  clinical  efficacy  of  the  PACS  system  in 
the  clinics: 

-  If  the  PCP  documented  his/her  own  interpretation  of  the  study;  and 

-  If  the  PCP’s  and  the  radiologist’s  interpretation  agreed  with  each  other.  (Figure  11) 


Figure  11:  Clinical  impact  of  PACS  of  NMHS  clinic  patients 


Pre-PACS  (%) 

Post-PACS  (%) 

p-value 

Cases  of  PCPs  who  documented  their 
interpretation  of  their  patients’  CXR 

28  pts  (47%) 

28  pts  (47%) 

1.00 

Cases  of  agreement  between  PCPs 
interpretation  and  the  radiologists’ 
interpretation 

25  pts  (89%) 

24  pts  (86%) 

1.00 

To  better  understand  Figure  11,  note  the  following  observation. 

1.  Less  than  half  (47%)  of  the  PCPs  documented  their  interpretations,  even  before  PACS 
was  implemented.  This  pattern  of  PCP  documentation  persisted  after  PACS,  which 
means  that  only  half  of  the  PCPs’  interpretations  could  be  assessed  for  agreement  with 
the  radiologists’  interpretation. 

2.  An  additional  interpretation  of  this  finding  is  that  the  implementation  of  PACS  did  not 
appear  to  change  the  PCP’s  behavior,  i.e.,  they  interpreted  their  films  at  the  same 
frequency,  regardless  of  PACS.  One  of  the  basic  premises  of  implementing  PACS  in  the 
clinics  is  to  provide  the  clinic-based  PCPs  with  the  radiologist’s  interpretation  during  the 
patient’s  visit.  As  noted  in  Figure  2,  the  turnaround  time  of  the  arrival  of  the  radiologist’s 
interpretation  on  the  patient’s  EMR  decreased  from  49.0  to  1.6  hours.  This  incredible 
improvement,  however,  did  not  change  the  frequency  of  the  PCPs  recording  their  own 
interpretations  on  the  EMR  (47%  pre  and  post  PACS).  And,  understandably,  it  did  not 
appreciably  change  the  agreement  between  the  PCP  and  radiologists  agreement  (89%  pre 
and  86%  post  PACS). 

3.  There  were  seven  cases  (3  pre-PACS  and  4  post-PACS)  in  which  the  radiologist’s 
interpretation  disagreed  with  the  PCPs’  interpretation.  Figure  12  describes  the 
differences  that  were  found  and  their  clinical  impact. 

It  is  not  possible  to  assess  the  actual  impact  on  the  PCPs  use  of  the  information  they  received. 
Based  on  their  comments  in  the  satisfaction  survey  (Figure  6),  it  appears  that  the  physicians  are 
using  the  radiologists’  interpretations  to  make  their  treatment  decisions. 


16 


Each  of  the  seven  cases  of  disagreement  between  the  PCP’s  and  the  radiologist’s  interpretation  of 
the  patients’  CXR  was  reviewed  by  the  clinic  system’s  quality  improvement  physician.  He 
determined  if  there  was  any  clinical  impact  on  the  patient’s  care  because  of  the  disagreement. 


Figure  12 


Pre-PACS 

Cases 

Case  Description 

Clinical 

Impact 

1 

The  PCP  interpreted  the  CXR  of  52  yo  male  with  coughing  and 
wheezing  as  a  “little  patch  of  pneumonitis”  and  prescribed 
antibiotics  for  bronchitis.  The  radiologist  interpreted  the  CXR 
as  “within  normal  limits.” 

None  -  would 
also  treat  these 
clinical  symptoms 
with  antibiotics 

2 

The  PCP  interpreted  the  CXR  of  16  yo  female  with  back  pain 
in  her  left  flank  area  as  “possible  costochrondral  tear”  and 
treated  her  with  a  mild  pain  medicine  and  an  anti-inflammatory 
agent.  The  radiologist  interpreted  the  CXR  as  a  “negative 
examination.” 

None-  would  also 
treat  the  pt’s  pain 
with  pain  med  & 
anti-inflammatory 
agents 

3 

The  PCP  interpreted  the  CXR  of  68  yo  female  with  cough, 
cold  and  congestion  “infiltrate  of  left  lower  lung.”  and 
prescribed  antibiotics  for  bronchitis.  The  radiologist 
interpreted  the  CXR  as  “within  normal  limits.” 

None  -  would 
also  treat  these 
clinical  symptoms 
with  antibiotics 

Post- 

PACS 

Cases 

Case  Description 

Clinical 

Impact 

1 

The  PCP  interpreted  the  CXR  of  57  yo  male  with  cough,  chest 
congestion  and  wheezing  as  “chronic  changes  and  light 
bronchial  cuffing”  and  prescribed  antibiotics  and  an  inhaler  for 
bronchitis.  The  radiologist  interpreted  the  CXR  as 
“unremarkable  chest.” 

None  -  would 
also  treat  these 
clinical  symptoms 
with  antibiotics 
and  an  inhaler 

2 

The  PCP  interpreted  the  CXR  of  6  yo  female  who  is  on  a 
return  visit  for  her  bad  cough  as  “a  suspicious  area  in  the  left 
base  for  an  infiltrate”  and  prescribed  a  different  antibiotic  and 
a  corticosteroid  for  possible  pneumonia.  The  radiologist 
interpreted  the  CXR  as  “and  infiltrate  in  the  lingular  segment 
of  the  left  upper  lobe  . . .  felt  to  represent  a  left  upper  lobe 
pneumonia.” 

None-  it  is 
difficult  to  make 
fine  anatomical 
distinctions  in  a 
small  child.  The 
interpretation  was 
similar  and  the 
treatment  would 
be  the  same. 

3 

The  PCP  interpreted  the  CXR  of  70  yo  female  with  lung 
congestion  as  “no  consolidations,  infiltrates  or  effusions”  and 
prescribed  antibiotics  and  cough  medicine  for  bronchitis.  The 
radiologist  interpreted  the  CXR  as  “a  slight  increase  in  the  size 
of  the  vague  nodular  opacity  in  the  right  upper  lobe.” 

Possible  -  would 
require  a  CT  of 
the  chest  to 
examine  nodular 
opacity. 

Follow-up :  PCP 
ordered  the  chest 

17 


CT  the  following 
day. 

4 

The  PCP  interpreted  the  CXR  of  9  yo  female  with  nighttime 
cough  and  shortness  of  breath  as  “clear”  and  prescribed 
antibiotics,  cough  medicine  and  a  corticosteroid  for  bronchitis. 
The  radiologist  interpreted  the  CXR  as  “peribronchial  wall 
change  consistent  with  bronchitis.” 

None  -  different 
terminology  but 
same  diagnosis 
and  treatment. 

While  PACS  increased  the  speed  in  the  radiologists’  return  of  their  interpretations,  it  did  not 
demonstrate  an  impact  on  clinical  outcomes  in  the  current  study.  In  retrospect,  we  may  have 
assessed  the  impact  too  soon  after  implementation  to  see  any  difference  in  PCP  behavior. 
Although  PCPs  indicated  they  are  looking  at  the  radiologists’  studies  before  making  their 
decisions  (per  satisfaction  comments)  they  did  not  appear  to  sign  off  on  the  radiologists’  notes 
significantly  earlier  than  before  (Figure  2).  This  may  indicate  that  many  of  the  PCPs  have  not  yet 
incorporated  this  new  technology  into  their  practices.  This  assessment  was  performed  shortly  (1- 
2  months)  after  PACS  was  implemented  in  the  clinics.  We  may  want  to  reassess  the  clinics  at  a 
later  interval,  like  after  one  year  of  experience  using  PACS. 

Post-Phase-3  Assessment  -  A  Review  of  the  PACS  Project  Conducted  in  April- June  2008 
Clinical  Outcomes  Assessment 

The  post-phase-3  clinical  outcomes  assessment  involves  both  the  hospital  setting  (phases  1  and 
2)  and  the  clinic  setting  (phase-3).  The  clinical  outcomes  assessment  for  the  first  two  phases 
involved  the  rapidity  of  diagnosis  and  treatment  of  community-acquired  pneumonia  in  the 
hospital  emergency  room  setting.  The  clinical  outcome  assessment  for  the  third  phase  involved 
the  concurrence  of  agreement  between  the  primary  care  provider’ s  (PCP)  interpretation  of  a 
CXR  obtain  at  the  clinic  and  the  radiologist’s  interpretation. 

Clinical  Impact  of  PACS  on  NMHS  Hospital  Patients 

Figure  13  pulls  together  pre-PACS  and  two  post-PACS  intervals,  the  2nd  being  the  most  recent, 
post-project  summary  period.  The  clinical  outcomes  measure  we  are  using  is  the  first  dose  of 
antibiotic  that  is  given  to  patients  who  enter  the  emergency  services  department  (ESD)  in  Tupelo 
and  at  the  community  hospitals  (CHs)  with  signs  and  symptoms  of  community  acquired 
pneumonia  (CAP).  The  diagnosis  of  CAP  is  made  upon  the  interpretation  of  the  CXR,  so  it  is  a 
good  clinical  process  marker  to  correlate  with  the  rapidity  of  radiological  interpretation.  The 
timing  of  the  first  dose  of  antibiotics  to  treat  CAP  is  also  correlated  with  the  clinical  outcomes  of 
CAP;  specifically,  higher  mortality  is  linked  to  delayed  antibiotic  administration  (over  8  hours). 
The  goal  for  first  dose  antibiotics  is  less  than  4  hours  and  this  is  used  as  a  “core  measure”  for 
CMS  and  Joint  Commission  Accreditation  of  hospitals.  Figure  13  shows  that  as  the  mean 
turnaround  time  of  all  radiological  study  interpretations  decreased,  so  did  the  time  to 
administration  of  the  first  dose  of  antibiotics. 
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Figure  13 


Facility 

PRE-PACS 

Oct-Dec  03  (Tupelo) 
April-Sent  04(CHs) 

1st  dose  ABx- 
Minutes  ( mean  study 
turnaround  time-hrs ) 

1st  POST-PACS 
Oct-Dec  04  (Tupelo) 
Auril-Seut  05(CHs) 

1st  dose  ABx- 
Minutes  (mean  study 
turnaround  time-hrs) 

2nd  POST-PACS 
Oct-Dec  07  (Tupelo 
and  CHs) 

1st  dose  ABx- 
Minutes  (mean  study 
turnaround  time-hrs) 

Tupelo  (Mean  Time) 

205  min 

(2.3  hrs) 

195  min  (1.0  hrs) 

163  min 

(0.9hrs) 

Eupora  (Median  Time) 

176  min 

(17  hrs) 

98  min  (1.3  hrs) 

118  min 

(1.3  hrs) 

Iuka  (Median  Time) 

NA 

(17  hrs) 

NA  (1.8  hrs) 

135  min 

(l.lhrs) 

Pontotoc  (Median 

Time) 

120  min 

(13  hrs) 

115  min  (0.5  hrs) 

93  min 

(0.1  hrs) 

West  Point  (Med  Time) 

120  min 

(9  hrs) 

120  min  (1.8  hrs) 

111  min 

(1.7hrs) 

Clinical  Impact  of  P ACS  on  NMHS  Clinic  Patients 

Forty  patient  visits  were  randomly  selected  from  two  of  the  rural  medical  clinics  that 
implemented  PACS  (20  charts  from  each  clinic  before  and  20  from  each  clinic  after  PACS).  The 
patients’  electronic  medical  records  (EMR)  were  reviewed  pre-PACS  implementation  (January 
2006)  and  post-PACS  implementation  (October  2007)  (Fig.  14).  In  the  pre-PACS  phase  there  is 
a  considerable  difference  (89%  for  Clinic- 1  and  50%  for  Clinic-2)  in  the  numbers  of  PCPs  who 
interpret  (and  document  their  inteipretation)  of  their  own  studies.  Once  PACS  was  implemented, 
fewer  PCPs  in  Clinic- 1  interpreted  their  own  studies  than  pre-PACS  (70%  vs.  89%).  The  Clinic- 
2  PCPs  interpreted  only  5%  of  their  own  studies  post-PACS  (vs.  50%  pre-PACS)  which  indicates 
that  PACS  did  have  an  impact  on  their  practice  (Figure  14).  Once  the  PCPs  became  accustomed 
to  receiving  the  radiologists’  interpretation,  they  no  longer  made  their  own  interpretation. 


Figure  14 


Pre  PACS 

Jan  06 

Clinic-1 

Pre  PACS 
Jan  06 
Clinic-2 

Post  PACS 
Oct  07 
Clinic-1 

Post  PACS 

Oct  07 

Clinic-2 

Cases  of  PCPs  who 
documented  their 
interpretation  of  their  patients’ 
CXRs 

17/19  (89%) 

10/20  (50%) 

14/20  (70%) 

1/20  (5%) 

Cases  of  agreement  between 
PCPs’  and  radiologists’ 
interpretation 

15/17  (88%) 

9/10  (90%) 

13/14  (93%) 

1/1  (100%) 

Over  the  pre  and  post  PACS  periods,  there  were  four  cases  of  disagreement  between  the  PCP’s 
and  the  radiologist’s  interpretations  for  a  9.5%  disagreement  (42  overall  cases).  The  Quality 
Director  for  the  clinic  system  (also  a  PCP)  reviewed  the  clinical  circumstances  of  the  four  cases 
of  disagreement  (Figure  15).  Despite  the  difference  between  the  radiologist  and  PCP’s 
interpretations,  our  physician  reviewer  did  not  identify  any  deficiency  in  care. 
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Figure  15 


Pre-PACS  Cases 

Case  Description 

Clinical  Impact 

1  (Clinic- 1) 

PCP  interpreted  the  CXR  of  70  yo  female  who 
presents  with  left  hip  pain  after  a  recent  fall  as 
“no  obvious  infiltrate,  bronchitis  changes.”  At 
the  time  of  the  office  visit  she  was  being  treated 
with  antibiotics  and  medrol  for  previous  (10  days 
prior)  c/o  increased  cough,  congestion,  rhinitis, 
generalized  myalgias  and  chills.  Pt  stated  her 
bronchitis-like  symptoms  were  not  improving  and 
PCP  prescribed  Rocephin  injection,  Biaxin  and 
continuing  Medrol  dosepak.  The  radiologist 
interpreted  the  CXR  as  “mild  infiltrate  or 
atelectasis  suggested  at  the  right  base.” 

No  change  in  overall 
approach. 

Reviewing  PCP  would 
have  prescribed  10  days 
of  antibiotic  rather  than 
a  7-day  course. 

2  (Clinic- 1) 

PCP  interpreted  the  CXR  of  83  yo  female  who 
presents  for  nursing  home  placement  as  having 
“some  mild  pulmonary  vascular  congestion.” 

The  radiologist  interpreted  the  same  film  as  “lung 
fields  appear  clear.” 

No  change 

The  PCP  commented 
on  the  CXR  but  did  not 
change  therapy  because 
of  it. 

3  (Clinic-2) 

PCP  interpreted  the  CXR  of  94  yo  female  with 
complaints  of  coughing  and  headache  as  “some 
signs  of  senile  emphysema”  and  treated  her  with 
oral  antibiotics  and  cough  syrup.  The  radiologist 
state  “no  new  infiltrate.” 

No  change  in  care. 
Levoquin  is  appropriate 
for  coronary  obstructive 
pulmonary  disease 
(COPD)  exacerbation. 

Post-PACS  Case 

4  (Clinic- 1) 

PCP  interpreted  the  CXR  of  64  yo  female  who 
presents  with  cough  productive  with  a  “bunch  of 
junk”  as  thickening  of  bronchial  walls  indicating 
inflammation.  The  PCP  oral  antibiotics  and 
Mucinex.  The  radiologist  inteipreted  the  CXR  as 
“lungs  are  well  expanded  and  clear.” 

None  -  would  also  treat 
these  clinical  symptoms 
with  antibiotics. 

►  Research  Area:  Identification  of  highly  effective  security  architecture  for 
networked  PACS  systems  that  can  be  replicated  for  Army  Medicine’s  networked 
medical  devices,  including  PACS 

BACKGROUND  regarding  phase-2  evaluation:  NMMC  contracted  Sword  &  Shield  Enterprise 
Security,  Inc.  (Sword  &  Shield)  to  conduct  a  network  vulnerability  assessment  and  penetration 
test  of  NMMC’ s  external  and  interned  networks.  The  assessment  examined  NMMC's 
implementation  of  the  Army  Security  Architecture  for  Medical  (ARSAM),  a  defense-in-depth 
network  security  architecture  for  FDA-approved  medical  devices.  NMMC  is  using  ARSAM  in  its 
deployment  of  the  PACS  within  its  hospital  information  system  network. 

Sword  and  Shield  Findings  from  May  2006 
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•  From  an  external  perspective,  the  overall  security  posture  of  the  NMMC  network  was 
deemed  to  be  satisfactory.  The  firewall  and  network  devices  provided  an  adequate  level 
of  protection  to  the  NMMC  internal  network,  and  PACS  network,  from  Internet-based 
threats. 

•  Flow  ever,  Sword  and  Shield  was  able  to  circumvent  the  ARSAM  protections  implemented 
at  NMMC  during  the  internal  phase  of  testing  by  taking  advantage  of  some  PACS  devices 
that  were  located  outside  of  the  protected  enclave  and  also  by  taking  advantage  of  other 
devices  on  the  network  which  were  not  patched  with  all  of  the  latest  security  patches  or 
system/applications  updates. 

Post-Phase-3  Assessment  -  A  Review  of  the  PACS  Project  Conducted  in  April- June  2008 
Post-Phase-3  Assessment  -  A  Review  of  the  PACS  Project  Conducted  in  April-June  2008 

NNMC  contracted  with  Sword  and  Shield  to  perform  a  network  vulnerability  assessment.  The 
assessment  was  performed  in  June  2008  and  Sword  and  Shield  produced  a  report  on  July  11, 
2008.  The  Executive  Summary  of  the  report  is  presented  and  NMMC’s  response  to  the  identified 
weaknesses  is  also  presented. 

Sword  and  Sheild’s  EXECUTIVE  SUMMARY  (July  11,  2008) 

Sword  &  Shield  Enterprise  Security,  Inc.  (Sword  &  Shield)  was  contracted  to  conduct  a  network 
vulnerability  assessment  (NVA)  and  penetration  test  (PT)  of  the  North  Mississippi  Medical 
Center  (hereinafter  referred  to  as  NMMC)  external  and  internal  networks.  The  assessment 
examined  NMMC’s  implementation  of  the  Army  Security  Architecture  for  Medical  (ARSAM);  a 
defense-in-depth  network  security  architecture  for  FDA-approved  medical  device.  NMMC  is 
using  ARSAM  in  their  deployment  of  the  Picture  Archive  and  Communications  System  (PACS) 
within  their  hospital  information  system  network. 

The  assessment  was  divided  into  two  phases.  First,  an  external  assessment  was  completed  under 
a  “zero  knowledge”  scenario  to  represent  a  “hacker’s  perspective.”  NMMC  provided  Sword  & 
Shield  consultants  with  only  a  range  of  public  IP  addresses.  Sword  &  Shield  consultant  targeted 
publicly-accessible  NMMC  assets  from  the  Internet  in  an  attempt  to  access  the  NMMC  network, 
and  subsequently  the  PACS  network.  While  the  assessment  team  was  unable  to  gain  access  to 
any  NMMC  systems  from  the  Internet,  high-risk  vulnerabilities  were  found.  Under  the  correct 
circumstances,  these  vulnerabilities  could  lead  to  the  remote  exploitation  of  a  system.  Secondly, 
upon  completion  of  the  external  assessment,  an  internal  NVA/PT  was  completed.  The  phase  took 
place  while  connected  directly  to  the  internal  NMMC  network  to  represent  an  “internal 
perspective.”  For  this  phase,  NMMC  provided  Sword  &  Shield  consultants  with  detailed  network 
information.  The  assessment  was  focused  on  the  assumption  that  a  person  with  malicious  intent 
had  connected  an  unauthorized  system  to  the  NMMC  internal  network  and  was  attempting  to 
gain  access  to  sensitive  data  within  the  PACS  network.  From  an  internal  perspective,  the  NMMC 
network  contained  deficiencies  that  led  to  the  complete  compromise  of  internal  systems  and, 
ultimately,  the  PACS  network. 
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The  PACS  network  is  currently  segmented  from  the  rest  of  the  internal  NMMC  network  through 
the  use  of  a  router  access  control  list  (ACL).  The  ACL  in  place  was  sufficient  in  keeping  normal 
traffic  within  the  NMMC  internal  network  from  reaching  the  PACS  network.  However,  the 
security  controls  provided  by  the  router  ACL  can  be  easily  circumvented  through  assuming  the  IP 
address  of  a  “permitted”  host,  compromising  a  “permitted”  host  and  using  it  as  a  proxy  for  access 
into  the  PACS  network,  or  directly  compromising  the  network  router  controlling  the  ACL. 
Vulnerabilities  on  the  internal  NMMC  network  enabled  Sword  &  Shield  consultants  to  both 
directly  compromise  the  network  router  controlling  the  PACS  ACL,  as  well  as  compromise 
“permitted”  hosts  that  had  access  to  the  PACS  network.  These  vulnerabilities  were  the  result  of 
missing  security  patches  and  system/application  updates,  blank  or  common  passwords,  and 
systems  or  applications  deployed  with  widely  known  default/insecure  settings. 

Based  on  the  analysis  of  the  results,  it  is  recommended  that  stateful  filtering  devices,  such  as  a 
firewall,  be  used  to  regulate  traffic  between  the  NMMC  internal  network  and  PACS  network 
segments.  Proper  configuration  would  allow  only  the  specific  services  needed  for  normal 
network  operation  to  pass  between  the  two  networks.  It  is  also  recommended  to  move  as  many  of 
the  devices  associated  with  the  PACS  network  into  the  protected  PACS  enclave.  Fewer  devices 
requiring  access  between  the  two  networks  will  ultimately  limit  the  number  of  attack  paths  into 
the  PACS  network. 

Development  and  implementation  of  formal  policies  and/or  procedures  is  also  recommended  to 
address  patch  management  and  configuration  management.  NMMC  already  deploys  an  internal 
patch  management  solution,  but  should  make  sure  that  it  includes  all  systems  within  the  NMMC 
network.  Patches  and  updates  to  be  applied  to  systems,  applications,  and  network  devices  on  a 
routine  basis. 

Finally,  it  should  be  noted  that  this  security  assessment  focused  solely  on  vulnerabilities  within 
the  NMMC  network  that  would  allow  an  attacker  to  access  the  PACS  network.  Based  on  the 
results  of  the  hosts  tested  under  this  limited  scope,  it  is  recommended  that  a  full  network 
vulnerability  assessment  and  penetration  test  be  performed  on  the  entire  NMMC  network  to 
address  other  potential  vulnerabilities. 

If  the  recommendations  contained  in  this  report  are  implemented  properly,  the  security 
posture  of  the  NMMC  network  will  be  significantly  improved. 


NMMC’s  Response  to  Sword  and  Shield’s  Findings  (July  11,  2008) 

NMMC's  implementation  of  the  ARSAM  security  design  was  unsuccessful  in  keeping  a 
malicious  user  from  gaining  access  to  the  PACS  network.  NMMC  will  continue  to  use  the 
Router  ACL  method  for  separating  the  PACS  network  from  the  remainder  of  the  NMMC 
network  as  it  is  successful  in  keeping  "normal"  traffic  from  within  the  NMMC  network  from 
reaching  the  PACS  network.  This  will  protect  the  PACS  network  from  unsophisticated  viruses, 
worms  and  malicious  software  which  might  infiltrate  the  NMMC  network.  NMMC's  imaging 
modalities,  which  are  more  vulnerable  to  such  attacks  due  to  a  lack  of  up-to-date  software  and 
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security  patches  from  vendors,  will  be  much  less  likely  to  be  compromised  from  malicious 
software  on  the  NMMC  network. 


NMMC's  failure  to  achieve  protection  from  an  individual  with  malicious  intent  is  due  in  large 
part  to  the  fact  that  NMMC's  PACS  network  is  integrated  with  many  systems  outside  of  the 
PACS  enclave:  it  is  a  part  of  the  NMMC  domain  and  uses  domain  authentication  for  user  level 
access;  it  uses  NMMC's  DNS  services;  and  it  is  integrated  with  NMMC's  HIS,  RIS,  and  interface 
engine,  all  of  which  are  located  outside  of  the  PACS  enclave.  NMMC  faces  a  situation  similar  to 
that  of  modality  vendors  in  that  it  can  not  immediately  apply  patches  or  updates  to  the  these 
systems  that  communicate  with  PACS.  Like  vendors,  NMMC  requires  time  to  thoroughly  test 
any  impact  the  patches  or  updates  might  have  on  our  production  environment.  This  testing 
requires  not  only  that  NMMC  test  the  PACS  system  in  regard  to  the  changes,  but  any  other 
systems  affected  as  well.  For  example,  any  changes  to  NMMC's  interface  engine  would  require 
validation  of  all  systems  integrated  through  it.  This  is  a  process  that  typically  takes  NMMC  six- 
to-eight  months  to  accomplish. 

The  Sword  and  Shield  evaluations  demonstrated  that  a  sophisticated  user  with  malicious  intent 
could  take  advantage  of  the  lack  of  timely  patches  to  these  external  systems  and  use  that 
advantage  to  circumvent  the  protections  provided  by  the  Router  ACL  method  of  separating  the 
two  environments. 

The  intent  of  the  ARSAM  security  architecture  with  respect  to  how  it  would  be  utilized  within 
the  DOD  environment  is  much  more  complex.  The  DOD  implementation  would  intend  to  protect 
not  only  the  PACS  devices,  but  more  specifically,  to  protect  the  remainder  of  the  DOD  systems 
from  any  user  or  device  that  resides  on  the  PACS  enclave.  Any  such  implementation  of  the 
ARSAM  architecture  would  require  an  enormous  amount  of  resources  dedicated  to  continuously 
monitoring  potential  for  compromise  in  addition  to  constant  patching  of  operating  systems  and 
commercial  software  packages  which  would,  in  turn,  require  continuous  testing  of  how  these 
patches  would  affect  involved  systems.  Not  unlike  the  situation  vendors  face  when  qualifying 
patches  for  devices  for  which  they  have  obtained  FDA  51  OK  clearance,  the  DOD  would  be 
challenged  to  perform  all  of  the  necessary  testing  required  before  updating  systems  with  patches. 
At  the  rate  with  which  security  patches  are  released,  it  would  be  difficult  to  maintain  such  an 
environment  without  an  enormous  amount  of  resources  and  effort. 

Project  Deviations 

NMMC-Hamilton  -  delayed  PACS  implementation  from  Phase-2  until  post  Phase-3: 

The  community  hospital  located  in  Hamilton,  AL  is  a  component  of  NMHS,  but  because  it  is 
located  in  a  different  state,  we  cannot  implement  some  programs  in  the  same  way  as  at  our 
Mississippi  community  hospitals.  The  major  technical  obstacle  was  the  cost  of  bandwidth  to 
the  Hamilton  hospital.  The  MS  community  hospitals  utilized  a  DS-3  line  which  is  45  megabytes. 
NMHS  spends  about  $4,000  per  month  for  line  charges  per  MS  community  hospital,  but  crossing 
the  state  line  into  Alabama  is  exponentially  higher,  making  it  impractical  to  implement  PACS. 
Hamilton  would  use  two  T-l  lines  which  would  be  only  3  megabytes.  And  with  the  low 
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bandwidth  the  PACS  service  would  be  too  slow  to  be  useful  (e.g.,  it  would  take  900  seconds  for 
a  CT  scan  with  385  images  to  arrive  at  the  radiologist’s  screen). 

To  solve  this  technical  problem  we  turned  to  Cisco  and  their  wide  area  application  services 
(WAAS).  WAAS  utilizes  multiple  technologies  to  optimize  bandwidth  across  a  WAN,  including 
the  following: 

•  Application  acceleration:  Cisco  WAAS  mitigates  latency  and  bandwidth  through  advanced 
protocol  optimizations,  including  read-ahead,  message  prediction,  and  caching. 

•  Throughput  optimization:  Cisco  WAAS  improves  behavior  of  transport  protocols  to  make  them 
more  efficient  in  WAN  environments. 

•  Bandwidth  optimization:  Cisco  WAAS  minimizes  the  transmission  of  redundant  data  patterns 
through  Cisco  WAAS  Data  Redundancy  Elimination  (DRE)  and  compression. 


In  testing  this  technology  we’ve  found  that  a  large  imaging  study  that  would  take  900  seconds  to 
come  across  the  first  time  (to  the  radiology  technician)  would  only  take  23  seconds  when  the 
radiologist  later  accessed  it. 

We’ve  had  to  invest  in  the  Cisco  WAAS  routers  in  order  to  take  advantage  of  this  technology. 
The  WAAS  technology,  however,  has  enabled  us  to  implement  PACS  in  Hamilton,  AL  using  two 
T-l  lines  and  at  what  will  be  an  affordable  monthly  rate.  We  also  plan  to  install  these  WAAS 
routers  at  the  rural  clinics  that  are  not  adjacent  to  the  rural  community  hospitals.  These  clinics 
use  DSL  or  cable  for  their  PACS  transmissions  and  the  routers  will  speed  up  the  transmissions. 

Reportable  Outcomes 

To  date,  no  manuscripts,  abstracts,  grant  applications,  patents,  etc.  have  been  submitted  that  are 
direct  results  of  this  project. 

Conclusions 

Implementation  of  PACS  technology  on  NMMC’s  campus  in  Tupelo,  four  community  hospitals 
and  15  clinics  has  yielded  information  of  potential  value  to  other  health  care  agencies  serving 
rural  regions: 

Summary  of  Findings 

Efficiency  -  faster  report  turn-around  and  better  use  of  radiologist  time 

•  PACS  technology  is  associated  with  dramatic  decreases  in  radiological  report  turn-around 
in  rural  hospital  and  rural  clinic  settings. 

•  PACS  technology  improved  the  radiologist’s  efficiency  in  reading  films  by  15%,  thereby 
enabling  the  existing  radiologist  staff  to  manage  an  increase  in  film  volume  by 
interpreting  more  studies  during  a  work  cycle. 
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Cost  -  cost  savings  depends  on  the  setting  ofPACS 

•  NMMC-Tupelo  campus  reduced  personnel  (film  librarian)  costs  by  $300,000  per  year  and 
this  level  has  remained  constant.  NMMC-Tupelo  reduced  film  costs  by  $450,000  per 
year  in  year-1,  but  by  year-4  this  savings  has  increased  film  savings  to  $600,000  per  year. 
NMMC-Tupelo  did  not  experience  an  increase  in  transmission  costs,  so  these  savings  are 
not  offset. 

•  All  four  community  hospital  sites  reduced  film  costs  by  an  average  of  $24,340  per 
hospital  per  year.  Two  of  the  community  hospitals  also  experienced  a  reduction  in 
personnel  (film  librarian)  costs  an  average  of  $1 1,000  per  hospital  per  year  after 
implementation  of  PACS. 

•  The  clinic  system’s  film  cost  savings  were  offset  by  their  increase  costs  in  IT  support  and 
transmission  costs.  This  results  in  a  22%  cost  increase  radiological  study  support  in  the 
clinic  system  because  ofPACS. 

Satisfaction  -  physicians  like  PACS 

•  Clinic-based  PCPs  and  referring  physicians  reported  greater  satisfaction  with  radiological 
services  following  introduction  ofPACS  capabilities. 

Clinical  Outcomes  -  ESD  administration  of  first  dose  antibiotic  to  CAP  patient  is  likely  the  best 
process  marker  on  the  clinical  impact  ofPACS. 

•  Community  hospital  data  (Figure  13)  support  the  hypothesis  that  the  use  of  PACS 
technology  increases  the  speed  with  which  antibiotics  are  administered  as  indicated  for 
CAP. 

•  The  low  usage  of  tPA  in  the  management  of  ischemic  stroke  makes  it  an  unreliable 
process  indicator  for  efficacy  of  PACS  in  the  management  of  stroke. 

•  The  implementation  of  PACS  in  the  clinic  system  did  affect  the  frequency  of  PCPS 
interpreting  their  patients’  CXRs.  PACS  reduced  PCPs  interpretation  of  CXRs  once  the 
PCPs  became  aware  of  and  comfortable  with  the  radiologists’  interpretation  timely 
arrival. 

Security  Architecture  Testing 

•  Initial  network  sensitivity  and  penetration  testing,  performed  in  May  2006,  revealed  that 
ARSAM  protections  implemented  at  NMMC  were  inadequate  and  that  additional 
precautions  needed  to  be  taken  in  the  security  system  architecture. 
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•  The  second,  and  final,  network  sensitivity  and  penetration  testing,  performed  in  July 
2008,  revealed  similar  findings  as  the  first  testing,  namely  that  although  it  was  difficult  to 
externally  penetrate  the  system,  there  was  complete  compromise  during  the  internal 
penetration  testing. 

•  NMMC  finds  that  the  major  ARSAM  security  impediment  is  the  timely  implementation 
of  security  patches.  Commercial  software  vendors  inconsistently  provide  security  patches 
and  because  of  the  impact  on  multiple  systems,  it  can  take  six-to-eight  months  to  test 
some  patches  before  they  can  be  installed 

•  NMMC  finds  that  it  difficult  to  maintain  the  ARSAM  architecture  without  considerable 
increase  in  its  current  resources  and  efforts. 


In  summary,  the  four  year  phase-in  of  PACS  technology  at  NMMC's  main  hospital  campus  in 
Tupelo,  five  community  hospitals,  and  15  clinics  has  yielded  mixed  results.  As  hypothesized,  the 
introduction  of  PACS  at  hospitals  and  clinics  comprising  NMHS  yielded  increased  efficiency  in 
radiological  services,  decreased  radiologic  staffing  (film  librarians),  improved  the  delivery  of 
first  dose  antibiotic  to  patients  with  CAP,  and  increased  satisfaction  of  primary  care  providers 
with  radiological  services.  In  contrast  to  expectations,  however,  it  did  not  appear  to  measurably 
improve  clinical  outcomes  for  people  with  ischemic  stroke.  Perhaps  most  suiprising  was  the 
variation  in  the  cost  savings  data:  while  PACS  was  associated  with  decreased  radiological 
expenses  at  NMMC-Tupelo  and  all  four  community  hospitals,  it  actually  increased  radiological 
expenses  by  22  percent  at  the  clinics.  Security  testing  found  that  the  system  has  not  responded 
satisfactorily  to  network  vulnerability  assessments  and  penetration  tests  conducted  by  an 
independent  vendor  (Sword  &  Shield). 

As  required  by  the  Research  Technical  Reporting  Requirements,  “So  what?”  Work  completed 
during  the  three  phases  of  PACS  implementation  and  research  at  NMMC  does  indeed  have 
implications  for  rural  hospitals/health  systems  and  for  the  military.  PACS  technology  has 
dramatically  increased  the  turn-around  time  for  radiological  images  and  increased  significantly  the 
number  of  “first  reads”  made  by  radiologists  rather  than  other  physician  specialists.  PACS 
technology  will  likely  be  instrumental  in  improving  physician  satisfaction  with  radiological  services 
at  other  rural  clinics  and  hospitals,  which  struggle  to  recruit  and  retain  qualified  health  care 
providers.  When  contemplating  implementation  of  PACS,  some  hospitals  may  be  able  to  project 
cost  savings  from  personnel  or  supply  (film)  budgets. 

The  results  of  network  sensitivity  and  penetration  testing  completed  at  NMMC  suggest  that  its 
implementation  of  ARSAM  did  enhance  the  protection  of  the  PACS  network  from 
unsophisticated  viruses  and  malicious  software,  yet  in  spite  of  a  progressive  information 
technology  department,  NMMC  finds  that  it  cannot  protect  against  a  sophisticated  user  with 
malicious  intent  who  had  gained  internal  access  to  our  network.  These  findings  would 
recommend  that  other  rural  hospitals  include  stringent,  formal  policies  and/or  procedures  to 
address  security  patch  management  and  configuration  in  the  implementation  of  their  PACS 
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systems  and  that  they  plan  for  a  large  increase  in  resources  required  to  keep  those  policies  and 
procedures  fully  implemented.  Test  results  regarding  security  patch  management  are  also  of 
significance  to  the  military's  efforts  to  deliver  radiological  services  from  remote,  noncombat  sites 
to  battlespaces  utilizing  existing  DOD  network  resources. 

The  health  system’s  experiences  will  be  used  to  formulate  recommendations  for  installation, 
implementation,  and  application  of  PACS  systems,  including  highly  effective  security 
architecture.  Data  collected  on  indicators  pertinent  to  clinical  outcomes  will  be  used  to 
document  the  impact  of  PACS  technology  in  civilian  applications. 

Appendices 

Not  applicable 
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